Congestion in InfiniBand Networks

The InfiniBand Architecture (IBA) is a relatively new industry-standard networking technology suited for inter-processor and I/O communication. One of the challenges facing IBA networks is how to deal with congestion. The most recent version of the IBA specification includes a Congestion Control Annex (CCA) that relies on end-to-end Explicit Congestion Notification (ECN) packet marking to resolve congestion with traffic injection rate throttling. However, setting the CCA parameters to control congestion in a stable and efficient manner requires some experimentation. There are alternatives to ECN that can be used to deal with congestion, including adaptive routing and Virtual Output Queuing (VOQ.) Due to the specific requirements of IBA, adaptive routing and VOQ are tricky to implement, but it can be done. Once implemented, adaptive routing can be effective at preventing congestion by routing around it, and VOQ can be effective at eliminating congestion spreading by bypassing it. However, these two techniques do not address the root cause of congestion as ECN does. This paper examines the advantages and disadvantages of each technique, and suggests that the best approach may be to combine the techniques, in order to both avoid congestion where possible, and recover from it when